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We study three dimensional systems where strong repulsion leads to an insulating state via sponta- 
neously generated spin-orbit interactions. We discuss a microscopic model where the resulting state 
is topological. Such topological 'Mott' insulators differ from their band insulator counterparts in 
that they possess an additional order parameter, a rotation matrix, that describes the spontaneous 
breaking of spin-rotation symmetry. We show that line defects of this order are associated with 
protected one dimensional modes in the strong topological Mott insulator, which provides a bulk 
characterization of this phase. Possible physical realizations in cold atom systems are discussed. 



I. INTRODUCTION 

Three dimensional topological insulators (TI) have re- 
cently raised considerable interest in both theory[l, 2, 3| 
and experiments 0, 01 for their nontrivial topological 
properties, which separate them from conventional band 
insulators. The hallmark of a TI is its exotic surface 
electronic properties. In particular, the strong topolog- 
ical insulator has an odd number of Dirac nodes on the 
surface of the system, which are stable against moderate 
perturbations that preserve time reversal symmetry 
Such a band structure cannot be realized in any two di- 
mensional system with time reversal invariance. There 
have been experimental realizations of these predictions 
in bismuth antimony [J Q alloys and in bismuth sele- 
nium 0, Q, which have been verified by angle resolved 
photoemission spectroscopy. 

In two dimensions, an analogous state is the quantum 
spin hall (QSH) phase, which has counter-propagating 
one dimensional edge modes. Such a state can occur 
with spin orbit interactions that preserve spin rotation 
symmetry about an axis, in which case the oppositely 
propagating modes carry opposite quantum numbers for 
this component of spinj^, It was shown in Ref . [lo| . 
that even in the absence of such spin rotation invariance, 
the counter-propagating modes remain protected by time 
reversal symmetry. The three dimensional weak topolog- 
ical insulator (WTI) can be obtained by stacking such 
2D QSH states. However, in order to realize the strong 
topological insulator, the spin rotation symmetry must 
be completely broken. 

The TI and QSH phases normally exist in the systems 
with strong spin-orbit interaction (SOI) that explicitly 
breaks spin rotation svmmetrv fSRS)[ll. [lol. 11 1. How- 
ever, as pointed out in Ref.[12| an extended Hubbard 
model on a 2D honeycomb lattice can have spontaneous 
SRS breaking and result in a QSH phase, with the right 
kind of repulsive interactions [l^. Spin rotation symme- 
try is only preserved about an axis n, which is spon- 
taneously chosen, leading to gapless Goldstone modes. 
This was termed a topological 'Mott' insulator - the sepa- 
ration of energy scales between the low lying magnetic ex- 



citations and the gapped charge excitations being typical 
of Mott insulators. We will also adopt this nomenclature 
although it must be noted that local moment physics, of- 
ten associated with Mott insulators, does not occur here. 
Subsequently, it was argued in [T^ that skyrmions of n 
carry charge 2e. 

Here, we consider the analogous problem of a three di- 
mensional system without bare spin orbit couplings, and 
full spin rotation symmetry, being driven into a topo- 
logical insulator state by strong interactions. The key 
difference from the two dimensional case, is that in order 
to realize the strong topological insulator, spin rotation 
symmetry must be completely broken. Hence the order 
parameter in this case is a rotation matrix R G 0(3), 
similar to superfluid Helium-3 A and B-phases, which 
leads to a richer set of topological textures. We de- 
scribe a microscopic model, an extended Hubbard model 
on the diamond lattice that, within a mean field treat- 
ment, leads to this phase. The order parameter supports 
a number of topological defects. In particular, a vortex 
like line defect occurs, but with a Z2 charge. This line 
defect in the strong TI is found to be associated with 
a pair of gapless fermionic excitations that travel along 
its length. These modes are topologically stable against 
moderate perturbations such as impurities and interac- 
tions as long as time reversal symmetry is intact. This 
is the main result of the paper - an analytical derivation 
is provided which relies on the properties of the Dirac 
equation on a two dimensional curved surface. 

We now contrast our results with other recent work. 
Similar exotic behavior also occurs in TIs, along crystal 
defects such as dislocations. Gapless fermionic excita- 
tions emerge there when a Z2 parameter formed by the 
product of the dislocation Burgers vector and three weak 
topological insulator (WTI) indices [3] is nonzero - which 
in principle can occur in both the weak and strong TI. 
In contrast, in the present paper, the fermionic modes 
along the line defect are solely determined by the more 
elusive strong index. They are absent in the case of the 
weak TI. Thus far, the characterization of the TI phase 
has relied on the surface behavior. This result provides a 
route to identifying the strong topological Mott insulator 
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via a bulk property. 



on next-neighbor sites. 



Similar modes have been identified propagating along 
a solenoid of tt flux, inserted into a strong topological 



insulator [15|. Here, the 2tt rotation of the electron spin 



around the line defect leads to a Berry's phase, provid- 
ing a physical realization of the tt flux. Analogous phe- 
nomena occur in the context of line defects in superfluid 
Hc3-Bil6i. 

The order parameter R admits a skyrmion like tex- 
ture, which is a point object in three dimensions (Shankar 
monopole). We find that in contrast to the skyrmion of 
the quantum spin Hall effect these are uncharged. 

In most solids where electron-electron interactions are 
important tend to have some degree of spin-orbit interac- 
tions - which will confine the defects. Hence, we propose 
realizations of this physics in optical lattices of ultracold 
atoms, utilizing molecules with multipole moments to ob- 
tain the proposed extended Hubbard models. The two 
dimensional version [l^] is found to be naturally realized 
with electric dipoles. Realizing the three dimensional 
case is more challenging, however molecules with elec- 
tric quadrupole moments confined in optical lattice can 
realize some of the key ingredients required. 

This paper is organized as the following: In the next 
section, we will present the order parameter manifold and 
the line modes' Z2 dependence on the winding number; in 
section III, we will justify our claim with numerical and 
analytical results; another texture Shankar monopole will 
be discussed in section IV; in section V, we will establish 
our model Hamiltonian on a diamond lattice and show 
the mean field stability of topological Mott insulating 
(TMI) phases; we give two possible experimental realiza- 
tions in cold atom systems in section VI; we conclude the 
main result of this paper in the last section. Hereafter we 
use a and r for the spin and sublattice degree of freedom, 
respectively. 



II. TOPOLOGICAL MOTT INSULATORS AND 
ORDER PARAMETER TEXTURES IN THREE 
DIMENSIONS 



In order to describe the Topological Mott insulating 
phase, we consider a concrete example in the following. 
In a subsequent section, we address the question of how 
such a phase may be microscopically realized. To con- 
trast the TMI phase with the regular topological insula- 
tor, consider the model Hamiltonians of a TI introduced 
in [l| . We consider nearest neighbor hopping (ty ) on the 
sites of a diamond lattice, and spin orbit induced hopping 



Hti 
Hhop 



Hhop + Hso 



yjCj (1) 



where c' 



^c|,c|^ is the electron creation operator 

and a is the spin pauli matrix. The spin-orbit interac- 
tion for a pair of second neighbor sites ij, depend on df^ 
(p — 1, 2) the two nearest neighbor bond vectors con- 
necting the second neighbor sites ij. The spin orbit in- 
teractions are thus determined by the crystal structure. 
Note, the SU(2) spin rotation symmetry is completely 
broken, which is required to realize the strong topologi- 
cal insulator in three dimensions. 

In contrast, in the TMI phase discussed here, the un- 
derlying Hamiltonian possesses full SU(2) spin rotation 
symmetry that is spontaneously broken. The order pa- 
rameter R then is a rotation matrix that describes the 
relative orientation between the real space coordinate 
system and the spin axes. The spin orbit term then takes 
the form: 



Hso 



{8Xso/a^ 



E4- 



Ri 



(2) 



There is one important difference between the TI and 
TMI: since in the latter R arises from symmetry break- 
ing it can vary spatially to give rise to a topologically 
nontrivial texture. To identify the topologically stable 
defects, we first note that the order parameter manifold 
is a three dimensional orthogonal matrix R G O (3). It 
can represent by i? = (ei, 62, 63), where are orthogo- 
nal unit vectors representing the basis vectors of the spin 
coordinate system. An example is shown in Fig. [TJ 

The target manifold of this order parameter is O (3) — 
SO (3) X Z2, and Z2 determines the chirality det ^ -R ^ = 
±1, or whether the rotation is proper or improper. Here- 
after we mainly focus on the continuous SO (3) part 
of the order parameter, for it has nontrivial homotopy 
groups in two and three dimensions. Then, each proper 
rotation can be described by the parameters (n, 6) , where 
n is the direction of the rotation axis and 9 G [0, tt] is the 
rotation angle around it. To visualize it we can map all 
proper rotation matrices to a solid ball with radius tt, 
where fi maps to the radial direction and rotation angle 
maps to the radius of the image point. Note a rotation of 
TT about n is the same as that of tt about — n, so opposite 
points on the spherical surface are identified. The re- 
sulting geometry is a three dimensional projective plane 
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III. ELECTRONIC STRUCTURE OF A LINE 
DEFECT: NUMERICAL AND ANALYTICAL 
RESULTS 

We study the electronic structure of the Z2 hue defect 
in the diamond lattice model of a topological Mott insu- 
lator discussed before. We choose the nearest neighbor 
hopping in three directions to be equal tij = t > Q while 
the fourth is different ty = t + St. A strong (weak) TI 
phase occurs \i 5t > Q {St < 0) pj. We choose an order 
parameter texture that incorporates line defects: 



FIG. 1: An illustration of order parameter R of a, nontriv- 
ial line defect in the x — y plane, z direction is translation- 
invariant. The hollow and solid arrow are the x and y axes of 
local coordinate, the z axis points out of the paper for proper 
rotation. 



We now discuss the topological defects of this order 
parameter space. The discrete Z2 symmetry breaking 
implied by the Z2 x SO{Z) order parameter leads to do- 
main walls in three dimensions. More interestingly, line 
defects also exist (we assume three spatial dimensions in 
the following). These can be captured by considering the 
order parameter along a closed curve in real space, which 
encircles the line defect. This defines a closed loop in the 
order parameter space and distinct classes of such closed 
loops correspond to the topological hue defects. There 
are two classes of closed loops for the 50(3) space de- 
scribed above. In addition to the trivial closed loop, that 
can be shrunk continuously to a point, there exists a non- 
trivial loop that connects the antipodal points (n, tt) and 
(— n,7r). Since these represent the same rotation, this is 
in fact a closed loop. Thus there is a nontrivial line defect 
characterized just by a Z2 topological charge. Technically 
TTi {SO (3)) = ^2 2l|. An example is shown in Fig.[T]with 
translational invariance along z direction although gener- 
ically, the line can be of arbitrary shape and direction. 
The electronic properties of such a hue defect is stud- 
ied in the following section - protected one dimensional 
modes that propagate along the defect are found in the 
case of the strong TMI, but not in the case of the weak 
TMI. We also note that since ttj {SO (3)) = 0, no non- 
trivial point defects exist in three dimensions. However, 
since tt^ {SO (3)) = Z, 'skyrmion' like textures (called 
Shankar monopoles [l8|) exist in three dimensions. In 
contrast to topological defects, they are smooth textures 
without a singular core. We investigate the electronic 
structure of these objects and find that they are neutral 
in the large size limit, in contrast to skyrmions in the two 
dimensional quantum spin hall state which carry charge 
2eM- 



(3) 

(cos {lip) sin {lip) 
—sin {lip) cos {lip) 
1 

where Ri depends only on the azimuthal angle of the 
atom connecting sites i and j. Note, only the parity 
of I is topologically stable. Here we study a system of 
24 X 24 with a maximally separated vortex {1 = 1) and 
anti- vortex {I = —1), and with periodic boundary con- 
dition in X — y plane and translational invariance in z 
direction. Note, time reversal symmetry is preserved by 
this Hamiltonian. 

For a St > strong TMI system with ^ = ±1, two 
pairs of conducting line modes are found in the bulk gap. 
These states' density profiles are strongly localized at the 
cores or the two defects (Fig. [2] top, shows only one of 
them for clarity). Therefore a Kramer pair of modes is 
localized along the thread of core. Given the particle hole 
symmetry that happens to be present in this model, they 
cross at zero energy (Fig. [2] bottom) . In contrast, these 
modes are absent in the cases of the weak TMI St < or 
if ^ = 0, ±2 in either type of TMI, and the bandstructure 
remains fully gapped. This is direct evidence that these 
Z2 dependent line modes within the bulk identify the 
strong TMI. 

An analytical argument for these modes can be devel- 
oped in several ways. We can derive these modes based 
on the 3D Dirac continuum limit of the Hamiltonian p|): 
however, below we consider deriving these modes using 
the known properties of surface states of strong topolog- 
ical insulators. Consider a bulk sample with a infinitely 
long cylindrical hole of radius R drilled through its cen- 
ter. We consider the low energy states on the cylindrical 
surface, both without and with a line defect inserted in 
the cylindrical hole Fig. [31 We show that in the limit 
of i? — > 0, a propagating midgap mode survives only 
when the defect is present; otherwise a fully gapped in- 
sulator results. Interestingly, the key ingredient here is 
the property of the two dimensional surface Dirac state, 
on a curved manifold. In contrast to the states of a 
particle on a ring, where the zero angular momentum 
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FIG. 2: Up: The density distribution of a midgap mode 
{k — 1.05, E = 0.25) in the a\,a2 plane, the mode is well local- 
ized at the 1 = 1 vortex, the other state {k = 1.05, E = —0.25) 
is well localized at the Z = — 1 antivortex; Down: Electronic 
spectrum of diamond lattice strong TMl along pz in the pres- 
ence of a pair of defects. The parameters used in the Hamil- 
tonian are t = 1.0, St = 1.0, and Ago = 0.125. 



eigenstate remains at low energy even when the ring ra- 
dius is shrunk to zero, here, the Dirac particle acquires a 
Berry's phase of tt on rotating around the cyhnder sur- 
face, which excludes the zero angular momentum state. 
Hence on shrinking the radius, no low energy states re- 
main, and the bulk insulator is recovered. However, if a 
topological defect of the spin-orbit coupling matrix is in- 
serted through the cylinder, an additional Berry's phase 
of TT is now acquired by the electrons. This ultimately 
results from the rotation of the electron's spin by 27r, on 
following the texture. With this additional phase, the 
zero angular momentum mode is allowed, and a low en- 
ergy propagating mode results when the cylinder radius 
is shrunk to zero. These are the topologically protected 
one dimensional modes in the core of the defect. Note, 
since we are establishing the presence of a topologically 
protected excitations, it is sufficient to use a simple Dirac 
dispersion for the surface states of the strong topologi- 
cal insulator, since a general surface state can always be 
adiabatically mapped to it. We first describe the surface 
Dirac hamiltonian in the presence of curvature, and ap- 
ply this to the case of a cylinder with a defect inserted 
through it. 

Dirac Theory on a Curved Surface: On a flat surface. 



spanned by the unit vectors rii, n2, the surface Dirac 
Hamiltonian for a strong TI can be taken as: 



(4) 



where we assume for simplicity that the spin lies in the 
same plane, with a relative angle of 9 to the momentum, 
hence ui = (fii cos 9 + fi2 sm9) ■ a and ctj — {n2 cos 6 — 
hi sm9) ■ a. Note, hi 9 — this reduces to H = p - a, as 
in In general, the a matrices involve both spin and 
sublattice degrees of freedom, but the essential physics is 
captured by taking them to be simply spin matrices. 

On a curved surface there should be additional terms 
due to the curvature. The effective Hamiltonian can 



be systematically derived [22|, as described in the Ap- 
pendix. Here we just present the result in the general 
case when the radii of curvature along the two tangent 
directions ni, n2 are Ri, R2 respectively (which are de- 
fined via {hi ■ p)hj = ihSijhi x h2/ Ri for i = 1, 2): 



-ffcurvod = • P + Cr27T.2 • P 

[sin 9 + i cos 9(hi x 712) • (?] 



-(— + — ) 
2 i?i i?2 



(5) 



We now apply this result to the problem of surface 
states on the cylindrical surface of radius R with axis 
along z and radius R with strong TMI outside and vac- 
uum inside (see Fig. [3]) . We use cylindrical coordinates 
z, (j), hence h^ = (0,0,1) and h^ = (— sin 0, cos 0, 0), 
hr = —hz X h(j, the two radii of curvature i?i = 00, R2 = 
R. For simplicity we consider 9 = 0. The effective Hamil- 
tonian in the absence of a defect is: 



iJo = {hz ■a)pz + {h^ ■a)p^ + 



n 



(6) 



where Pz = —ihdz and = —i^d^. We can solve for 
the energies Hq^ = Etj} by first performing the unitary 
transformation ifj = Uz{4>)ip' where Uz{4>) — e"*'*'^^/^. 
Note, since Uz{(t) + 27r) = —Uz{<f>), the new wavefunc- 
tions ip' satisfy antiperiodic boundary conditions. The 
transformed Hamiltonian Hq = PzO^z + P0Cj^ has eigen- 
vectors tp' = e*'^^e™'^x, where x is a fixed spinor. The 
energy eigenvalue E then satisfies: 



El{k) =h^{k^ + T?/R^) 



(7) 



now, due to the antiperiodicity of the -0', we require rL+^ 
to be integer. Hence, En{k) in equation[7]above, all corre- 
spond to massive Dirac dispersions whose mass increases 
as i? ^ 0. 

We now consider introducing a texture in the order 
parameter. A strength '1' is readily introduced by the 
spin rotation [/^ (/(/>). The Hamiltonian then is: 



Hi = Ul{l4>)HoUz{l<p) 



(8) 



The eigenstates ip of Hi can be obtained by the unitary 
transformation ip = Uz{[l — l\(f>)ip' , and the transformed 
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zJLR^ IV. SHANKAR MONOPOLE IN THREE 




DIMENSIONS 




FIG. 3: Illustration for the model on an effective Hamiltonian 
on a curved surface we study. The radius of the cylinder is R 
with strong TMI outside and vacuum inside, pz and p^p are 
momentum along z and the azimuthal direction, respectively. 
8 is the angle between a' and p. 



Hamiltonian for the wavefunctions is identical to Hq 
above. The energy eigenvalues are then given by EqnlT] 
However, the crucial difference is that the single valued- 
ness property of the wavefunction now requires 7i+ to 
be an integer. Hence, for odd values of I, when the topo- 
logically nontrivial defect is present, an n = solution 
is allowed. The dispersion of this mode is Eo{k) — ±k, 
hence there is a up and down moving mode, that survives 
at low energies even when i? — > 0. 

The physical picture is: on the surface of a cylinder 
the momentum p^p is quantized according to boundary 
condition and the inter-level spacing is proportional to 
R^^. Also, because of the spin-momentum relation, when 
electron circles about the line defect it picks up a Berry 
phase of (Z -I- 1) tt when the vortex winding number is 
I. When the total phase is an integer multiple of 2tt 
it enforces periodic boundary condition, otherwise anti- 
periodic boundary condition is applied and the quantized 
Pip will miss the origin. As we shrink the cylinder ra- 
dius to zero, all quantized will diverge except p^p = 0. 
Therefore, with time reversal symmetry a Kramers pair 
of line modes exist when I is odd and its dispersion only 
depends on pz- Also, the distinction between the strong 
TMI and weak TMI can be made clear by the number of 
surface modes. For a weak TMI, there are an even num- 
ber of surface modes, leading to an even number pairs 
of line modes with the above reasoning. However, they 
are unstable against inter-node scattering. In contrast, a 
strong TMI always has an odd number pairs of surface 
modes. 



The line vortex is not the only nontrivial texture in 3D. 
The other texture we study is the Shankar monopole, 
a mapping S3 SO (3) characterized by the homo- 
topy classification tts (50 (3)) = Z 18]. Imagine an 
identical phase faraway from the monopole, where ma- 
trix R (n, if) is independent of f, we can identify the 
infinity of real space thus obtain S3. As an spe- 
cific example here the R matrix connecting p and t? 
is R (f, r) = exp {iO (r) f • , where f and r are the 
directional vector and distance from the origin to the site 
linking i and j , and J are classical rotation generators in 
three dimensions. The rotation axis h = f, and the rota- 
tion angle is (r), a function at the origin and smoothly 
increases to 21tt {I G Z) at infinity. The homotopy class 
TTa {SO (3)) is described by integer I, suggesting the base 
manifold wraps the target manifold I times. It is pro- 
tected against any continuous deformation. This is a 
zero dimensional defect so the localized states should be 
localized charge at the monopole, if any. 

However, numerical results show this topological de- 
fect does not carry localized states even in the strong 
TMI phase. We studied a single monopole at the cen- 
ter of a 32"^ unit cell system. To treat the large system 
size we found a way to sidestep a complete diagonaliza- 
tion of the spectrum. Instead, we estimate the bound- 
aries of the energy eigenstates using ARPACK (Arnoldi 
Package), and shift the spectrum so that all states be- 
low (above) the band gap are at negative (positive) en- 
ergies. Then, we only need to look at the difference in 
the number of negative and positive energy eigenvalues, 
to determine the monopole charge. This can be done 
via an efficient LDL^ factorization, where the Hamilto- 
nian is factorized into a lower triangular matrix L and a 
diagonal matrix D. The entries of the diagonal matrix 
preserves the sign of the eigenvalues, but not their mag- 
nitude. Counting the number of positive and negative 
eigenvalues is then readily accomplished. While smaller 
system sizes sometimes show charged monopoles, at the 
largest sizes, they are found to be neutral. We conclude 
that the Shankar monopole texture does not carry charge 
in the TMI phase. 



V. TOPOLOGICAL MOTT INSULATOR IN A 
MICROSCOPIC MODEL - EXTENDED 
HUBBARD MODEL ON THE DIAMOND 
LATTICE 

We now discuss the question of realizing the 3D TMI 
phase beginning with a microscopic model with full spin 
rotation symmetry. We consider an extended Hubbard 
model on the diamond lattice within mean field theory. 
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FIG. 4: (Color Online) A 3D plot of diamond lattice. Each 
unit cell contains two sublattices, denoted by A and B re- 
spectively, which each forms an fee crystal. The repulsive in- 
teractions V2 and V3 between the second neighbor and third 
neighbor are shown. They are both between the atoms of the 
same sublattice. 



As alvifays, the results of such a mean field treatment 
should be treated with caution especially since strong in- 
teractions are involved. Nevertheless, we use this analyti- 
cally tractable approach to obtain a range of parameters 
where the TMI phase is stabilized over the other obvi- 
ous candidate phases - the disordered semi-metal, anti- 
ferromagnetic insulator (or spin density wave: SDW) and 
the charge density wave (CDW). Realizing the 3D TMI 
which completely breaks spin rotation symmetry sponta- 
neously, requires, in mean field theory, further neighbor 
repulsion (between second than third nearest neighbors) 
as well as a small antiferromagnetic coupling between 
second neighbors, as shown in Fig. [5] 

We now discuss the details of this mean field treatment. 
The model Hamiltonian we study is an extended Hub- 
bard model on a 3D diamond lattice at half filling(Fig.[4l) 



Cja + h.C. 



/ViiPiPj + J S Si ■ Sj 



Without repulsive interaction the system is in a semi- 
metal phase with gapless excitations along lines in the 
Brillouin zone ('Dirac lines') and a vanishing DOS at 
the Fermi level. We turn on interactions and investigate 
possible phases including the Topological Mott Insula- 
tor, second and third nearest neighbor charge density 
wave (CDW) insulators, and nearest-neighbor spin den- 
sity wave(SDW) insulator. Note, the diamond lattice is 
composed of a FCC Bravais lattice, plus a two site basis 
that forms the two sublattices. The natural SDW phase 
has alternating spin densities on two sublattices, result- 
ing from the effective antiferromagnetic coupling from 
nearest neighbor hopping and U . Note, this is a g = 
order, i.e. preserves lattice translations. In contrast, the 
likely CDW orders resulting from second and third neigh- 
bor repulsion break translation symmetry within each 
sublattice. The phases can be mapped to a 3D Ising 
model on an FCC lattice [l^, and both the second and 
the third nearest neighbor CDWs has the same charge 
density distribution for two sublattices but nonuniform 
distribution from unit cell to unit cell. Finally, in the 
TMI phase one develops second nearest-neighbor correla- 
tions (cJ^Cjo-') ~ i^aa' ■ -R • i^Ai X dj^ that mimic the spin 
orbit interaction and breaks the SU (2) symmetry com- 
pletely. Within the mean field approximation we solve 
the ground state energy for each phases in the following, 
and the resulting phase diagram with fixed J — 0.3t is 
shown in Fig.O Note there is a TMI phase in the center. 

We now discuss the mean field energetics of these 
phases in more detail. 

(i) Semimetal: For relatively weak interactions, the 
semimetal phase arising from the nearest neighbor hop- 
ping model on the diamond lattice remains stable due to 
the vanishing density of states at the Fermi energy. This 
phase retains all symmetries of the Hamiltonian. 

(ii) Spin Density Wave In the limit of large onsite re- 
pulsion U the SDW phase with opposite spin on the two 
sublattices is stabilized. More precisely, if U — 2AJ 3> V2, 
the SDW phase becomes energetically favorable. Define 
the order parameter 9 



where t is the nearest neighbor hopping strength, J is 
the second nearest-neighbor antiferromagnetic coupling 
strength between spins Si = c\$Ci, and Vij = V2 for 
second nearest-neighbor repulsion, Vij — Vz for third 
nearest-neighbor repulsion while U is the onsite repulsion 
strength. All of these operate within the same sublattice 
as can be seen from figure Fig. [D For simplicity, we as- 
sume no nearest neighbor interaction: Vi = 0; however, 
as long as Vi (or fourth- nearest neighbor repulsion V4) is 
not so large that a nearest-neighbor charge density wave 
becomes energetically favored, they can be included but 
will be irrelevant to our mean field results. We neglect 
further neighbor interactions, riic, = c\^Cia is the number 



operator on site i for spin cr, and pi = Uif ■ 



HI ■ 



■1. Note 



the Hamiltonian has full SU (2) spin rotation symmetry. 



The ground state energy per unit cell is calculated using 
mean field approximation: 

EsDW = ^ + (I - 12j) X - -^5]7(C/-24J)^x + 4|<(fc)| 



Here x — cos^29. x — denotes the semimetal phase. 
t{k) — tJ2 e*'' *" , tn — 1, 2, 3, 4 are the vector from one 

Tl=l 

atom to its nearest neighbor. L is the number of points 
along each direction within the first Brillouin Zone. 

(Hi) Charge Density Wave In the limit of strong fur- 
ther neighbor repulsion V2, V3, a CDW is expected. The 



7 



problem with just the density repulsion can be mapped to 
the FCC lattice Ising model. In that context it is known 
that V2 will favor a what is called a Type III CDW phase, 
while V3 will favor a type II CDW phase (T^, described 
below. The type III CDW phase has Neel state in the 
(100) plane and frustrated arrangement between neigh- 
boring planes which leads to a Z2 degeneracy per plane; 
the type II CDW phase can be described as a combina- 
tion of four independent simple cubic CDWs. The CDW 
phase is important despite the frustration. As a mat- 
ter of fact, in Ref.[l3| the second nearest neighbor CDW 
phase, which the authors neglected, will dominate the 
large V2 region and the Quantum Anomalous Hall phase 
can be realized only with inclusion of U. To suppress the 
CDW phases, we choose the ratio of V3/V2 to be 1/2 so 
the system is close to the transition between type II and 
type III CDW phases[i3. Hereafter we fix V3 = V2/2, 
and point out that the phase diagram is similar without 
V3 but the TMI phase will generally occur at a larger U 
region. 

Assuming inversion symmetry and SRS arc intact, we 
define order parameter p for type III CDW phase 



H = 



Cfe 



negative of these. We decouple the Hamiltonian 

k 

h.c.) - ^ 2 (I/2 - J) Ixl sin (k ■ 4) 

os9i-j (^c\a'' ® TzCkj + sm% (^e'"^''cla^ ® r^Cfe -f h.c.^ 

where the summation is over the two occupied bands. 
dij are the vector from on site to its six second nearest 
neighbors (one for each opposite pair). We then obtain 
the ground state energy per unit cell: 

Etmi ^ J + 24 iV2- J) \x\^ 



L3 



E 



\t{k)\' + 4{V2-jf\x\' 



where Xij = [sinOijCoscjjij , sinOij sirujii 



1 



P 



1 - 



where footnote 1 and 3 arc on the first sublattice of two 
neighboring unit cells and footnote 2 and 4 are on the 
second sublattice and a is the spin index. The energy 
per original unit cell is 

EcDW = 3V2P^ + U(1- p") 12 



^Jlya' + 1-921' + I53I' ± I52I' + (5233* +52*53) 



where gi = {W2-U/2)p, 52 = t [l + e'^'"^^ j , 

^ ^gifc aa _j_ gifc a3~j ^ ^^^^ ^^iQ lattice vectors for the 

FCC lattice. The momentum summation is over i^/2 
points of the unit cell doubles. The CDW instability is 
signalled by a non-zero p. 

(iv) Topological Mott Insulator More importantly, simi- 
lar to the QSH phase in 2D ij], at intermediate couplings 
the TMI phase is favored, with order parameters: 



COsOr 

sin9j,je' 



where i,j are second nearest neighbors. Our mean field 
ansatz assumes for simplicity that all Xij have the same 
magnitude but their directions are arbitrary and de- 
scribed by the angles 9ij and ipij. Note Xij — ~Xji for 
Hermiticity. Lattice translation, rotation and inversion 
symmetries are also considered to be intact. This implies 
that the order parameters on the other sublattice are the 



It is straightforward to see that the energy only de- 
pends on Ixl and the relative angles between Xij- Under 
global rotation to all Xij energy remains invariant 
and directly leads to an SU (2) degeneracy. 

This is not a TMI phase in the strict sense, since 
there are dirac nodes at the Fermi level. However, 
an arbitrarily small distortion of the lattice will intro- 
duce anisotropy of nearest neighbor hopping strengths 

tdist (fc) = trie*'' '" and an effective mass that leads to 

a gap of size Stf]\. In the simplest case that one near- 
est neighbor hopping strength is different from the other 
three tdist (k) = < (fc) + St, a stronger bond i5i > will 
lead to a strong TMI phase while a weaker bond St < 
will lead to a weak TMI phase. 

Phase Diagram: For each phase, we search for the 
global minimum with respect to its order parameters and 
compare different phases. Numerical evaluation of ener- 
gies was done on a Brillouin zone with 40 x 40 x 40 A: space 
points. For simplicity we present the phase diagram with 
a fixed next nearest antiferromagnetic coupling strength 
J = 0.3t, and V3 — O.5V2 (Fig. [5]). The semimetal phase 
exists at small U and V2 region; the CDW phase occurs 
at large V2; the SDW phase occurs at large U. Most im- 
portantly, there is a TMI phase in the center of the phase 
diagram. 

This TMI phase has second nearest neighbor corre- 
lation similar to that arising from spin orbit interac- 
tions in the Fu-Kane-Mele model on the same lattice, 
except, of course for an arbitrary SU(2) spin rotation [H, 

Xij ^ R ■ (^di X dj^ for each second nearest neighbor 
pair i and j, where di and dj are nearest neighbor bond 

vectors connecting this pair of sites, and R is an arbi- 
trary constant three dimensional rotation matrix. 
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FIG. 5; Phase diagram for an extended Hubbard model on 
3D diamond lattice. The phase transitions from semimetal to 
SDW, CDW and TMI are all second order transitions. Other 
parameters are V3 = V2/2 and J = 0.3t. The system size is 
L = 40 for calculation. 

If we further increase J the stability of the TMI phase 
is enhanced and it now a wider parameter range. How- 
ever, at still larger values, a different TMI phase, that 
breaks lattice symmetries is realized via a continuous 
transition. However, since this occurs in the very large 
U regime, where mean field theory may not be accurate, 
we do not present further details of this phase. 



VI. TOWARDS EXPERIMENTAL 
REALIZATIONS 

An experimental realization of the TMI phase must 
contend with two challenges. First, the system should 
have weak intrinsic spin-orbit coupling, but strong in- 
teractions. Next, the further neighbour repulsion should 
be substantial compared to the nearest neighbor inter- 
actions. We believe these difficulties can be overcome in 
cold atom system, where intrinsic spin orbit couplings are 
irrelevant, if particles with electric multipole moments 
are confined to optical lattice sites. We first discuss a 
two dimensional example involving electric dipoles, for 
which a fairly definite experimental setup can be con- 
structed. Although the phase realized here is two dimen- 
sional and does not break spin rotation symmetry com- 
pletely (a U(l) spin rotation remains unbroken), it illus- 
trates how the necessary ingredients can be assembled. 
Subsequently we discuss ideas for realizing the three di- 
mensional TMI, the main subject of this paper, using 
electric quadrupole moments. 

2D Case: Electric Dipoles on a diamond lattice layer 
Dipole-dipole interactions between hetero-nuclear polar 
molecules, such as Rbgy and K40 have already been shown 
to be strong [23|. Consider a fermionic spin 1/2 molecule. 



with an electric dipole moment (which is independent of 
the spin) confined to the sites of an optical lattice. We 
note here that the diamond lattice has a special prop- 
erty that if the dipole-moment is along the [100] di- 
rections, then the nearest neighbor interaction Vi van- 
ishes. Thus, the second nearest neighbor interaction 
V2 becomes dominant. However, the difficulty is that 
within the twelve second nearest neighbors, only interac- 
tions between neighbors within a plane perpendicular to 
the dipole moment are repulsive. This problem can be 
solved if we restrict the molecules within a two dimen- 
sional (111) layer of the diamond lattice (still contain 
both sublattices and essentially two layers of triangular 
lattices) , as the sites circled in Fig. |31 Then if the dipole 
moment is perpendicular to the plane all possible nearest 
neighbor interactions are repulsive. 

We solve for the mean field phase diagram of this 
model, as was done previously for the 3D case. Note, 
since the lattice is essentially the honeycomb lattice, this 
is essentially the model studied in Ref. 12]. There exists 
a 2D-TMI phase at the center of the U — V2 phase dia- 
gram (Fig. [nj . Note this phase diagram differs from the 
same model in Ref.|12|] which has an extended 2D-TMI 
phase. This is because we also allow for the second near- 
est neighbor CDW that the authors neglected. Though 
frustrated, this order will dominate at large V2. 

The resulting TMI phase in our case has a second 
nearest neighbor correlation resembling that of a quan- 
tum spin Hall state and SU(2) spin rotation symmetry is 
only broken down to U(l). The resulting order parame- 
ter wiU be SU (2) /U (1) = S2 instead of SU (2). Since 
TTi {S2) = there are no point topological defects - how- 
ever skyrmions acquire a charge 2e in this phase. Note, 
the parameters in the phase diagram seem rather accessi- 
ble for dipolar molecular systems in this setup, assuming 
that the onsite U, which results from a combination of 
dipolar and microscopic interactions, is not too large. 

3D TMI: Electric quadrupoles on a distorted diamond 
lattice. We now discuss some ideas for realizing the 3D 
TMI. While these are not as straightforward as the ones 
discussed earlier, we nevertheless offer them as one av- 
enue that presents itself at the current time. Another 
possible origin of repulsive interaction is the quadrupole- 
quadrupole interaction. One can show that for two paral- 
lel uniaxial quadrupoles (quadrupole tensors are identical 
and diagonal), the interaction is 

E = (3 - iOcos^e + 35cos^9) 

where Q is the electric quadrupole moment, r is the dis- 
tance between two quadrupoles and 6 is the angle be- 
tween the quadrupole symmetry axis and the direction 
between the quadrupoles. 

From this expression if cos^O ~ 0.742 the quadrupole 
-quadrupole interaction will vanish. Therefore, assume 
the quadrupole moment is along the crystal unit cell z 
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FIG. 6: Phase diagram for an extended Hubbard model on 2D 
diamond lattice layer including both sublattices (essentially 
honeycomb lattice). Note its diflFerence from Ref . . The 
TMI is limited to the center of the phase diagram and only 
partially breaks the SU (2) SRS. The system size is L = 40 
for calculation. 



VII. CONCLUSION 

We have argued that appropriate repulsive interactions 
can induce a spontaneous spin rotation symmetry break- 
ing state, the topological Mott insulator, where spin orbit 
couplings are induced by interactions. 

In addition to exotic surface states, line defects of the 
order parameter are found to carry protected one dimen- 
sional modes along their length in the strong TMI, which 
provides a bulk signature of this phase. Also, potential 
experimental directions towards creating these phases in 
cold atom system are discussed. An interesting open 
question is whether the form of the spin-orbit interac- 
tions near a line defect, and hence these protected line 
modes, can be realized by suitably modifying the atomic 
structure in a strong topological band insulator. 
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axis to retain as many symmetries as possible, if we elon- 
gate the lattice along one direction to make the near- 
est neighbor satisfy this condition, we will obtain a sys- 
tem where second nearest neighbor interaction domi- 
nates. This gives c/a ~ 2.40. Note the distortion 
brings no change to the Hamiltonian we started with. 
Within this lattice, the ratio between the repulsive in- 
teraction from the second nearest neighbor out of the 
quadrupole perpendicular plane and that from those in 
plane is E±/Ez = 0.945. The V2 anisotropy is reasonably 
small, thus we believe the physics we discussed with an 
isotropic V2 is unchanged. However, in this arrangement, 
V3 is rather small. 

To give an reasonable estimate of the quadrupole 
strength necessary to drive the system into a TMI phase, 
we notice the typical nearest neighbor hopping strength 
in an cold atom system is of order 10~* ^ 10~^K lim- 
ited by the cooling temperature and the typical lattice 
dimension is about the laser wavelength 0.5 x 10~^to. 
This leads to a quadrupole of 10~^^ ~ 10~^''e • m^, e is 
unit charge. This is rather large but maybe realizable in 
multi-electron molecules. Also, the critical quadrupole 
moment can be further reduced by lowering the temper- 
ature or using lasers with shorter wavelength. Finally, a 
moderate second nearest neighbor antiferromagnetic cou- 
pling may result from second nearest neighbor hopping 
super-exchange effect. We leave for future work construc- 
tion of a more realistic setting that can realize the 3D 
TMI phase. 



APPENDIX: EFFECTIVE HAMILTONIAN ON A 
CURVED TI SURFACE 

It is known that the effective Hamiltonian for a TI sur- 
face has the form of a Dirac equation: H = p ■ S when 
the spin and momentum are parallel and H — h - {p x a) 
when they are perpendicular, where n is the normal di- 
rection of the surface. Both the momentum and spin are 
confined to the surface. More generally, when the angle 
between spin and momentum is 9 the effective Hamilto- 
nian is: 

H = [hi ■ fjcosO + 712 ■ Ssind) {hi ■ p) 
+ {712 ■ ScosO — hi ■ asinO) (n2 • p) 

where hi and h2 are orthogonal directions in plane. 

One would tends to apply the same Hamiltonian to 
a curved surface. However, here we claim the effective 
Hamiltonian on an arbitrary shaped TI surface is: 

H = (hi ■ acosO + 77,2 • asinO) (hi ■ p) 
+ (n2 • fJCOsO — hi ■ asinO) (n2 • p) 

+Ki^ + i^)""'+?^""'^^ {k^^) '''''' 

where hi and h2 are tangent vectors of the surface with 
radius of curvature Ri and i?2, respectively, and 713 is 
the normal vectors. 6 is the constant angle between spin 
and momentum in the corresponding flat surface effec- 
tive theory. Note when the surface is flat, the last two 
terms vanish and the Hamiltonian goes back to the one 
describing the flat surface mode. 
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This Hamiltonian can be systematically derived from 
the inclusion of the connection form for a curved 



space [2l|, |22|, but an alternative method we used is 
to ensure hermiticity and anticommutation relation 
{H,a -77.3} = since the spin is in the surface 
plane [isj. with the help of the relations: {fii-pjUj — 
ihSijUs/Ri, {fii ■ p) — —ihfii/Ri for i = 1,2, and 

(a ■ Aj (a ■ B^j = A- B + ia ■ (^Ax By 

The additional terms arising from the space curvature 
are canceled by the inclusion of the connection form. 
Thus, the above effective Hamiltonian well describes a 
curved TI surface. 
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